Precision Recall

Multi-class Precision-recall

Please see below for a Venn schematic for illustrating Precision-Recall and:

This is a subset of observed fate percentages we have:

what we tune

We need to binarize the matrix for giving each lineage, one or more class labels as ground truths when constructing a confusion-matrix and deriving evaluation metrics such as:

1. Specificity, Sensitivity >> ROC;
2. Precision, Recall >> PR-cruve; F1 score etc.

We denote \(\tau\) as the lowest percentage of observed cell label when we assign a lineage such label.

For example, in case \(\tau\) = 0.2 & 0,4,

• The 1st highlighted lineage will be labeled as only neu anyway
• The 2nd will be labeled as both neu; mo and only neu respectively

Ery	Meg	Mast	Baso	Eos	Neu	Mo	Ccr7_DC	pDC	Lymphoid
0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0
0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
0.0	0.0	0.0	0.1	0.1	0.8	0.0	0.0	0.0	0.0
0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
0.0	0.0	0.0	0.0	0.0	0.7	0.3	0.0	0.0	0.0
0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0

Precision-Recall in our context

Even in the context of multi-class precision-recall, the calculation of precision-recall will be performed first within each class, in our case, each fate.

That being said, once we set the threshold \(\tau\), the number of recall denominator is fixed.

Then when we decrease the threshold \(p\) for labeling a lineage as a specific fate or not based on simulated percentage, precision denominator increase, the numerator (white intersection), thus recall should either stay unchanged or increase. But as a ratio, precision can change in a non-monotonic way.

What micro-averaging does is offerring a metric just like average pearson correlation we tried before, as shown below:

\[ \begin{aligned} & \mathrm{Precision_{ Neu}} = \frac{A_{Neu}}{B_{Neu}} \\ & \mathrm{Precision_{ Mo}} = \frac{A_{Mo}}{B_{Mo}}\\ & \mathrm{Precision_{ Micro}} = \frac{A_{Neu} + A_{Mo}}{B_{Neu} + B_{Mo}} \end{aligned} \]

Please see below for a set of precision-recall curves for two models when setting \(\tau = 0.4\)

Here, we will see how generally increase tau will harm the average precision but we are always doing better.